Enhanced Expressive Power and Fast Training of Neural Networks by Random Projections

Random projections are able to perform dimension reduction efficiently for datasets with nonlinear low-dimensional structures. One well-known example is that random matrices embed sparse vectors into a subspace nearly isometrically, known as the restricted isometric property in compressed sensing. In this paper, we explore some applications of deep neural networks. We provide expressive power fully connected networks when input data or form smooth manifold. prove number neurons required approximating Lipschitz function prescribed precision depends on sparsity manifold and weakly vector. The key our proof stably set subspace. Based fact, also propose new network models, where at each layer first projected onto by projection then standard linear connection non-linear activation applied. way, parameters significantly reduced, therefore training can be accelerated without too much performance loss.